OIDC token retry under concurrent deploys + cloudExtras.topologySpreadConstraints#329
Merged
Conversation
The keyless-signing OIDC token was fetched with a single HTTP GET and a hard 10s client timeout. When several stacks that share one image deploy concurrently from the same push, each process races for a token and the GitHub Actions OIDC endpoint intermittently times out, surfacing "requesting OIDC token: context deadline exceeded" and failing the deploy. Retry the request with exponential backoff and full jitter (4 attempts, 1s->8s) to decorrelate concurrent callers, replace the fixed client timeout with a per-attempt context deadline (20s), and abort the loop immediately when the parent context is cancelled (returning ctx.Err() unwrapped). Only retry transient failures (transport/timeout, 408, 429, 5xx); fail fast on 4xx and on a malformed or empty 200 body. Attempts and per-attempt timeout are overridable via SC_OIDC_TOKEN_REQUEST_ATTEMPTS / _TIMEOUT. Sharing one token across the racing processes would be strictly better but is a workflow-level change outside this package; this is the in-repo defense. Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
Add a topologySpreadConstraints field to cloudExtras so a stack can spread its replicas across nodes/zones. On GKE Autopilot this is the cost-correct way to achieve node-level HA: pod anti-affinity forces a 0.5 vCPU minimum request per pod (Autopilot rejects pods below it), while topology spread is not subject to that minimum, so replicas spread at their normal request. The constraint is threaded through both cloudExtras->DeploymentConfig converters (kubernetes-native and GKE Autopilot) and into the pod spec. Conversion is split into a pure normalize step (defaults + validation: maxSkew>=1 default 1, whenUnsatisfiable default DoNotSchedule, minDomains >=1 and only with DoNotSchedule, labelSelector defaults to the deployment's own pods) and a mechanical Pulumi mapping, so the defaulting/validation is unit tested without resolving Pulumi outputs. Threading guards cover both converters since they copy cloudExtras field-by-field and have drifted apart before. To force a second node on Autopilot use whenUnsatisfiable: DoNotSchedule with minDomains: 2; ScheduleAnyway only expresses a preference and lets the scheduler bin-pack replicas onto one node. Signed-off-by: Dmitrii Creed <creeed22@gmail.com>
Cre-eD
requested review from
Laboratory,
smecsia and
universe-ops
as code owners
June 17, 2026 08:13
Semgrep Scan ResultsRepository:
Scanned at 2026-06-17 08:13 UTC |
Security Scan ResultsRepository:
Scanned at 2026-06-17 08:14 UTC |
smecsia
approved these changes
Jun 17, 2026
universe-ops
approved these changes
Jun 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two independent reliability/scheduling improvements, kept as two commits for review and rollback:
1.
fix(security): retry GitHub Actions OIDC token request under loadKeyless-signing acquired the GitHub Actions OIDC token with a single HTTP GET and a hard 10s client timeout. When several stacks that share one image deploy concurrently from the same push, each process races for a token and the OIDC endpoint intermittently times out (
requesting OIDC token: ... context deadline exceeded), failing the deploy.contextdeadline (20s) that still honours parent-context cancellation (sleepsselectonctx.Done()).SC_OIDC_TOKEN_REQUEST_ATTEMPTS/SC_OIDC_TOKEN_REQUEST_TIMEOUT.Acquiring one token per job and sharing it via
SIGSTORE_ID_TOKEN(already short-circuited) would also help, but that's a workflow-level change outside this repo; this is the in-repo defense.2.
feat(k8s):cloudExtras.topologySpreadConstraintsAdds a
topologySpreadConstraintsfield tocloudExtrasso a stack can spread its replicas across nodes/zones.On GKE Autopilot this is the cost-correct alternative to pod anti-affinity for node-level HA: Autopilot enforces a 0.5 vCPU minimum request on pods that use pod (anti-)affinity (and rejects pods below it), whereas topology spread is not subject to that minimum — replicas spread at their normal request.
cloudExtras→DeploymentConfigconverters (kubernetes-nativeToKubernetesRunConfigand GKE AutopilotToGkeAutopilotConfig) and into the pod spec.normalize(defaults + validation) and a mechanical Pulumi mapping:maxSkewdefault1(must be ≥1);whenUnsatisfiabledefaultDoNotSchedule(must beDoNotSchedule|ScheduleAnyway);minDomainsonly valid withDoNotSchedule;labelSelectordefaults to the deployment's own pods.Tests
pkg/security: retry success/5xx/429/timeout/exhaustion, fail-fast on 4xx, no-retry on malformed/empty 200, parent-cancel-during-backoff abort, audience-URL building, env short-circuit.pkg/clouds/...:normalizedefaults/validation table, no-input-mutation, multiple constraints; threading guards on both converters (the k8s-native and Autopilot paths, which have drifted apart historically);ConvertDescriptordeserialization.All touched packages:
go build,go vet,go test,gofmtclean.Rollout
After release, consumers bump their pinned SC version to pick up both fixes. The topology-spread change is opt-in (no behavior change unless a stack sets the field).